Simulation models for aggregated data meta-analysis: Evaluation of pooling effect sizes and publication biases.

Simulation studies are commonly used to evaluate the performance of newly developed meta-analysis methods. For methodology that is developed for an aggregated data meta-analysis, researchers often resort to simulation of the aggregated data directly, instead of simulating individual participant data from which the aggregated data would be calculated in reality. Clearly, distributional characteristics of the aggregated data statistics may be derived from distributional assumptions of the underlying individual data, but they are often not made explicit in publications. This article provides the distribution of the aggregated data statistics that were derived from a heteroscedastic mixed effects model for continuous individual data and a procedure for directly simulating the aggregated data statistics. We also compare our simulation approach with other simulation approaches used in literature. We describe their theoretical differences and conduct a simulation study for three meta-analysis methods: DerSimonian and Laird method for pooling aggregated study effect sizes and the Trim & Fill and precision-effect test and precision-effect estimate with standard errors method for adjustment of publication bias. We demonstrate that the choice of simulation model for aggregated data may have an impact on (the conclusions of) the performance of the meta-analysis method. We recommend the use of multiple aggregated data simulation models to investigate the sensitivity in the performance of the meta-analysis method. Additionally, we recommend that researchers try to make the individual participant data model explicit and derive from this model the distributional consequences of the aggregated statistics to help select appropriate aggregated data simulation models.

Registered reports and replications: An ongoing Journal of School Psychology initiative.

Recent psychological research suggests that many published studies cannot be replicated (e.g., Open Science Collaboration, 2015). The inability to replicate results suggests that there are influences and biases in the publication process that encourage publication of unusual-rather than representative-results, and that also discourage independent replication of published studies. A brief discussion of the ways in which publication bias and professional incentives may distort the research literature in school psychology is contrasted against the importance of replications and preregistration of research (i.e., registered reports) as self-correcting mechanisms for research in school psychology. The limitations of current practices, coupled with the importance of registered reports and replications as self-correcting mechanisms, provide the context for this ongoing initiative in the Journal of School Psychology. Processes for manuscript submission, review, and publication are presented to encourage researchers to preregister studies and submit replications for publication.

Publication bias in psychology: A closer look at the correlation between sample size and effect size.

Previously observed negative correlations between sample size and effect size (n-ES correlation) in psychological research have been interpreted as evidence for publication bias and related undesirable biases. Here, we present two studies aimed at better understanding to what extent negative n-ES correlations reflect such biases or might be explained by unproblematic adjustments of sample size to expected effect sizes. In Study 1, we analysed n-ES correlations in 150 meta-analyses from cognitive, organizational, and social psychology and in 57 multiple replications, which are free from relevant biases. In Study 2, we used a random sample of 160 psychology papers to compare the n-ES correlation for effects that are central to these papers and effects selected at random from these papers. n-ES correlations proved inconspicuous in meta-analyses. In line with previous research, they do not suggest that publication bias and related biases have a strong impact on meta-analyses in psychology. A much higher n-ES correlation emerged for publications' focal effects. To what extent this should be attributed to publication bias and related biases remains unclear.

Influence of study characteristics, methodological rigour and publication bias on efficacy of pharmacotherapy in obsessive-compulsive disorder: a systematic review and meta-analysis of randomised, placebo-controlled trials.

QUESTION: We examined the effect of study characteristics, risk of bias and publication bias on the efficacy of pharmacotherapy in randomised controlled trials (RCTs) for obsessive-compulsive disorder (OCD). STUDY SELECTION AND ANALYSIS: We conducted a systematic search of double-blinded, placebo-controlled, short-term RCTs with selective serotonergic reuptake inhibitors (SSRIs) or clomipramine. We performed a random-effect meta-analysis using change in the Yale-Brown Obsessive-Compulsive Scale (YBOCS) as the primary outcome. We performed meta-regression for risk of bias, intervention, sponsor status, number of trial arms, use of placebo run-in, dosing, publication year, age, severity, illness duration and gender distribution. Furthermore, we analysed publication bias using a Bayesian selection model. FINDINGS: We screened 3729 articles and included 21 studies, with 4102 participants. Meta-analysis showed an effect size of -0.59 (Hedges' G, 95% CI -0.73 to -0.46), equalling a 4.2-point reduction in the YBOCS compared with placebo. The most recent trial was performed in 2007 and most trials were at risk of bias. We found an indication for publication bias, and subsequent correction for this bias resulted in a depleted effect size. In our meta-regression, we found that high risk of bias was associated with a larger effect size. Clomipramine was more effective than SSRIs, even after correcting for risk of bias. After correction for multiple testing, other selected predictors were non-significant. CONCLUSIONS: Our findings reveal superiority of clomipramine over SSRIs, even after adjusting for risk of bias. Effect sizes may be attenuated when considering publication bias and methodological rigour, emphasising the importance of robust studies to guide clinical utility of OCD pharmacotherapy. PROSPERO REGISTRATION NUMBER: CRD42023394924.

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics.

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Outcome Reporting Bias in Clinical Trials Researching Disease-Modifying Therapy in Patients With Multiple Sclerosis.

BACKGROUND AND OBJECTIVES: Outcome reporting bias occurs when publication of trial results is dependent on clinical significance, thereby threatening the validity of trial results. Research on immunomodulatory drugs in multiple sclerosis has thrived in recent years. We aim to comprehensively examine to what extent outcome reporting bias is present in these trials and the possible underlying factors. METHODS: We identified clinical trials evaluating the efficacy and safety of immunomodulatory drugs in patients with multiple sclerosis (MS) registered in ClinicalTrials.gov after September 2007 and completed before the end of 2018. Information about study design, type of funding, and primary and secondary outcome measures was extracted from the registry. Timing of registration in relation to study initiation and subsequent amendments to the planned outcomes were reviewed. Publications related to these trials were identified in several bibliographic databases using the trial registration number. Registered primary and secondary outcomes were recorded for each trial and compared with outcomes in the publication describing the main outcomes of the trial. RESULTS: A search of ClinicalTrials.gov identified 535 eligible registered clinical trials; of these, 101 had a matching publication. Discrepancies between registered and published primary and secondary outcomes were found in 95% of the trials, including discrepancies between the registered and published primary outcomes in 26 publications. Forty-four percent of the published secondary outcomes were not included in the registry. A similar proportion of registered and nonregistered reported primary efficacy outcomes were positive (favoring the intervention). Nonindustry-funded and open-label trials in MS were more prone to selective primary outcome reporting, although these findings did not reach statistical significance. Only two-thirds of the trials were registered in ClinicalTrials.gov before the trial start date, and 62% of trials made amendments in registered outcomes during or after the trial period. DISCUSSION: Selective outcome reporting is prevalent in trials of disease-modifying drugs in people with MS. We propose methods to diminish the occurrence of this bias in future research.

Publishing clinical trial results in plain language: a clash of ethical principles?

Plain language resources (PLR) are lay summaries of clinical trial results or plain language summaries of publications, in digital/visual/language formats. They aim to provide accurate information in jargon-free, and easy-to-understand language that can meet the health information needs of the general public, especially patients and caregivers. These are typically developed by the study sponsors or investigators, or by national public health bodies, research hospitals, patient organizations, and non-profit organizations. While the usefulness of PLR seems unequivocal, they have never been analyzed from the perspective of ethics. In this commentary, we do so and reflect on whether PLR are categorically advantageous or if they solve certain issues but raise new problems at the same time. Ethical concerns that PLR can potentially address include but are not limited to individual and community level health literacy, patient empowerment and autonomy. We also highlight the ethical issues that PLR may potentially exacerbate, such as fair balanced presentation and interpretation of medical knowledge, positive publication bias, and equitable access to information. PLR are important resources for patients, with promising implications for individual as well as community health. However, they require appropriate oversight and standards to optimize their potential value. Hence, we also highlight recommendations and best practices from our reading of the literature, that aim to minimize these biases.

Plain language resources (PLR) are a way to make medical research information easier for everyone to understand.They can be summaries of clinical trial results, articles, or presentations. PLR can also be made as videos, brochures, or infographics.They can help patients understand their health better and take care of themselves. However, there are some things to be careful about.PLR may only report the good results and not mention the negative ones, which could be biased.Also, some people with disabilities or who don't speak the language well might have a hard time understanding PLR.To make sure PLR are helpful and fair, there should be standard guidelines for how they are made and shared. This will make sure that PLR are useful and don't cause any problems.

A systematic review and meta-analysis on the effects of ill health and health shocks on labour supply.

BACKGROUND: Several studies have explored the effects of ill health and health shocks on labour supply. However, there are very few systematic reviews and meta-analyses in this area. The current work aims to fill this gap by undertaking a systematic review and meta-analysis on the effects of ill health and health shocks on labour supply. METHODS: We searched using EconLit and MEDLINE databases along with grey literature to identify relevant papers for the analysis. Necessary information was extracted from the papers using an extraction tool. We calculated partial correlations to determine effect sizes and estimated the overall effect sizes by using the random effects model. Sub-group analyses were conducted based on geography, publication year and model type to assess the sources of heterogeneity. Model type entailed distinguishing articles that used the standard ordinary least squares (OLS) technique from those that used other estimation techniques such as quasi-experimental methods, including propensity score matching and difference-in-differences methodologies. Multivariate and univariate meta-regressions were employed to further examine the sources of heterogeneity. Moreover, we tested for publication bias by using a funnel plot, Begg's test and the trim and fill methodology. RESULTS: We found a negative and statistically significant pooled estimate of the effect of ill health and health shocks on labour supply (partial r = -0.05, p < .001). The studies exhibited substantial heterogeneity. Sample size, geography, model type and publication year were found to be significant sources of heterogeneity. The funnel plot, and the trim and fill methodology, when imputed on the left showed some level of publication bias, but this was contrasted by both the Begg's test, and the trim and fill methodology when imputed on the right. CONCLUSION: The study examined the effects of ill health and health shocks on labour supply. We found negative statistically significant pooled estimates pertaining to the overall effect of ill health and health shocks on labour supply including in sub-groups. Empirical studies on the effects of ill- health and health shocks on labour supply have oftentimes found a negative relationship. Our meta-analysis results, which used a large, combined sample size, seem to reliably confirm the finding.

Metabolic dysfunction-associated steatotic liver disease and the risk of mortality in individuals with type 2 diabetes: a systematic review and meta-analysis.

The systematic review aimed to assess the risks of metabolic dysfunction-associated steatotic liver disease (MASLD) on all-cause and cause-specific mortality in patients with type 2 diabetes (T2DM). EMBASE and MEDLINE were searched from inception to June 2022 for observational studies examining the relationship between MASLD and the risk of mortality among T2DM patients. Meta-analysis was conducted using random-effects models with hazard ratios (HRs) to quantify the risk of mortality. A total of 5877 articles were screened, and ultimately, 12 eligible studies encompassing 368 528 T2DM patients, with a median follow-up of 8.9 years (interquartile range, 4.7-14.5), were included. Our analysis revealed a significant association between MASLD and an increased risk of all-cause mortality in T2DM patients [HR 1.28; 95% confidence interval (CI), 1.05-1.58; I 2  = 90%]. Meta-regression analyses did not show significant effects of mean age, mean BMI, and percentage of smokers, hypertension, and hyperlipidemia on the association between MASLD and the risk of all-cause mortality. However, we found that MASLD was not significantly associated with mortality related to cardiovascular diseases (HR 1.05; 95% CI, 0.82-1.35; I2  = 0%) or cancer (HR 1.21; 95% CI, 0.41-3.51; I 2  = 79%) among patients with T2DM. No publication bias was observed. This comprehensive meta-analysis provides substantial evidence supporting a significant association between MASLD and an increased risk of all-cause mortality among the T2DM population. These findings underscore the potential benefits of screening for MASLD in T2DM patients, aiding in the early identification of high-risk individuals and enabling risk modification strategies to improve survival.

P-hacking in meta-analyses: A formalization and new meta-analytic methods.

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.

Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials.

BACKGROUND: Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials. METHODS: The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles. Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article. RESULTS: The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint-article RCTs were included in the analysis. The median (interquartile range) delay between preprint and journal article was 121 (73-187) days, the median sample size was 150 (71-464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as 'some concern' for 80% of RCTs. We found that 81 preprint-article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found. CONCLUSIONS: Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power.

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Publication bias in otorhinolaryngology meta-analyses in 2021.

INTRODUCTION: One concern in meta-analyses is the presence of publication bias (PB) which leads to the dissemination of inflated results. In this study, we assessed how much the meta-analyses in the field of otorhinolaryngology in 2021 evaluated the presence of PB. METHODS: Six of the most influential journals in the field were selected. A search was conducted, and data were extracted from the included studies. In cases where PB was not assessed by the authors, we evaluated the risk of its presence by designing funnel plots and performing statistical tests. RESULTS: Seventy-five systematic reviews were included. Fifty-one percent of them used at least one method for assessing the risk of PB, with the visual inspection of a funnel plot being the most frequent method used. Twenty-nine percent of the studies reported a high risk of PB presence. We replicated the results of 11 meta-analyses that did not assess the risk of PB and found that 63.6% were at high risk. We also found that a considerable proportion of the systematic reviews that found a high risk of PB did not take it into consideration when making conclusions and discussing their results. DISCUSSION: Our results indicate that systematic reviews published in some of the most influential journals in the field do not implement enough measures in their search strategies to reduce the risk of PB, nor do they assess the risk of its presence or take the risk of its presence into consideration when inferring their results.

Posttraumatic stress and delay discounting: a meta-analytic review.

Delay discounting-the extent to which individuals show a preference for smaller immediate rewards over larger delayed rewards-has been proposed as a transdiagnostic neurocognitive process across mental health conditions, but its examination in relation to posttraumatic stress disorder (PTSD) is comparatively recent. To assess the aggregated evidence for elevated delay discounting in relation to posttraumatic stress, we conducted a meta-analysis on existing empirical literature. Bibliographic searches identified 209 candidate articles, of which 13 articles with 14 independent effect sizes were eligible for meta-analysis, reflecting a combined sample size of N = 6897. Individual study designs included case-control (e.g. examination of differences in delay discounting between individuals with and without PTSD) and continuous association studies (e.g. relationship between posttraumatic stress symptom severity and delay discounting). In a combined analysis of all studies, the overall relationship was a small but statistically significant positive association between posttraumatic stress and delay discounting (r = .135, p < .0001). The same relationship was statistically significant for continuous association studies (r = .092, p = .027) and case-control designs (r = .179, p < .001). Evidence of publication bias was minimal. The included studies were limited in that many did not concurrently incorporate other psychiatric conditions in the analyses, leaving the specificity of the relationship to posttraumatic stress less clear. Nonetheless, these findings are broadly consistent with previous meta-analyses of delayed reward discounting in relation to other mental health conditions and provide further evidence for the transdiagnostic utility of this construct.

A case study evaluating the effect of clustering, publication bias, and heterogeneity on the meta-analysis estimates in implant dentistry.

Meta-analyses may provide imprecise estimates when important meta-analysis parameters are not considered during the synthesis. The aim of this case study was to highlight the influence of meta-analysis parameters that can affect reported estimates using as an example pre-existing meta-analyses on the association between implant survival and sinus membrane perforation. PubMed was searched on 7 July 2021 for meta-analyses comparing implant failure in perforated and non-perforated sinus membranes. Primary studies identified in these meta-analyses were combined in a new random-effects model with odds ratios (ORs), confidence intervals (CIs), and prediction intervals reported. Using this new meta-analysis, further meta-analyses were then undertaken considering the clinical, methodological, and statistical heterogeneity of the primary studies, publication bias, and clustering effects. The meta-analyses with the greatest number and more homogeneous studies provided lower odds of implant failure in non-perforated sites (OR 0.49, 95 % CI = [0.26, 0.92]). However, when considering heterogeneity, publication bias, and clustering (number of implants), the confidence in these results was reduced. Interpretation of estimates reported in systematic reviews can vary depending on the assumptions made in the meta-analysis. Users of these analyses need to carefully consider the impact of heterogeneity, publication bias, and clustering, which can affect the size, direction, and interpretation of the reported estimates.

Publication Bias and Selective Outcome Reporting in Randomized Controlled Trials Related to Rehabilitation: A Literature Review.

OBJECTIVE: To investigate the rate of registered protocols published as research papers as a measure of publication bias, and the concordance rates of the primary outcomes between research protocol and published papers as a measure of selective outcome reporting bias in randomized controlled trials (RCTs) related to rehabilitation. DATA SOURCES: Protocols related to RCTs were extracted from electronic databases, the University Hospital Medical Information Network (UMIN), International Standard Research Clinical Trial Number (ISRCTN), ClinicalTrials.gov, and MEDLINE. Published papers were retrieved from MEDLINE. STUDY SELECTION: The inclusion criteria were as follows: (1) initial registration (UMIN, ISRCTN, ClinicalTrials.gov) within the designated period; (2) published as a paper from a research protocol in MEDLINE (PubMed); and (3) written in English or Japanese. The search period was from January 1, 2013, to December 31, 2020. DATA EXTRACTION: The outcome of this study was set as the rate of published papers that were consistent with the extracted research protocol and the concordance rate between the primary outcomes in published papers and in protocols. The concordance rate of the primary outcomes was evaluated by checking whether the description in the research protocol matched that in the paper's abstract and main text. DATA SYNTHESIS: Out of the 5597 research protocols registered, only 727 were published (13.0%). The concordance rates of the primary outcomes were 48.7% and 72.6% in the abstract and main text, respectively. CONCLUSIONS: This study revealed major discrepancies between the number of research protocols and published papers, and difference of description regarding the primary outcomes in published papers which were already defined in the research protocols.